Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there been limited work that studies the interaction between audio visual modalities cross-modal self-supervision. This article (1) investigates via face reconstruction to guide representations; (2) proposes an audio-only approach representation learning; (3) shows a multi-task combination proposed is beneficial richer features more robust noisy conditions; (4) self-supervised pretraining can outperform fully supervised training especially useful prevent overfitting on smaller sized datasets. We evaluate our learned representations discrete emotion recognition, continuous affect recognition automatic recognition. existing methods all tested downstream tasks. Our results demonstrate potential feature suggest joint leads informative